A Dynamic Extension and Data Migration Method Based on PVFS

With the development of the big data, The traditional file system can no longer meet the demand of High Performance Computing and Big Data. Parallel file systems are getting more and more popular in High Performance Computing. As a typical parallel file system, PVFS has been widely used in big data computing area in recent years. However with the increasing of computing scale, there exist the needs to dynamic extend data nodes, which PVFS does not support at present. This paper put forward a dynamic data node extension method as well as the subsequent data migration algorithm based on PVFS. The algorithm first adds a new data node automatically and transparently. After that, the algorithm finds out the most loaded data node in the original file system using a new load evaluation method and transfer the data into the newly added data node to mitigate the imbalance of the system. The experimental results show that our dynamic data node extension method could improve the performance of PVFS and reduce the probability of hot point effectively.